Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Storage]Mount cached mode #4369

Open
wants to merge 62 commits into
base: master
Choose a base branch
from
Open

[Storage]Mount cached mode #4369

wants to merge 62 commits into from

Conversation

landscapepainter
Copy link
Collaborator

@landscapepainter landscapepainter commented Nov 15, 2024 •
edited
Loading

Tested (run the relevant ones):

  • Code formatting: bash format.sh
  • Check if mountpoints remain mounted after restarting by running sky stop and sky start:
    • pytest tests/test_smoke.py::test_aws_storage_mounts_with_stop --aws
    • pytest tests/test_smoke.py::test_gcp_storage_mounts_with_stop --gcp
    • Manually running sky stop and sky start with comp_test.yaml to see if all the MOUNT_CACHED remains storage remains mounted.
  • Set of comprehensive manual tests with comp_test.yaml:
    • launch tests:
      • sky launch comp_test.yaml --cloud aws -y
      • sky launch comp_test.yaml --cloud gcp -y
    • sky launch two instances mounting same backend storage. Create new file on mountpoint on instance 1, and check if it appears shortly in instance 2.
    • managed spot jobs tests:
      • sky jobs launch comp_test.yaml --use-spot --cloud aws -y
      • sky jobs launch comp_test.yaml --use-spot --cloud gcp -y
    • launch on docker containers:
      • sky launch comp_test.yaml --cloud aws --image-id docker:continuumio/miniconda3:latest -y
      • sky launch comp_test.yaml --cloud gcp --image-id docker:continuumio/miniconda3:latest -y

comp_test.yaml:

file_mounts:
  #### S3 tests ####
  /s3-sky-managed-copy:
    name: s3-sky-managed-copy
    source: ~/sky_logs
    store: s3
    mode: COPY

  /s3-sky-managed-mount:
    name: s3-sky-managed-mount
    source: ~/sky_logs
    store: s3
    mode: MOUNT

  /s3-sky-managed-mount-cached:
    name: s3-sky-managed-mount-cached
    source: ~/sky_logs
    store: s3
    mode: MOUNT_CACHED

  /s3-external-private-by-user:
    source: s3://s3-external-private-by-user-dy
    mode: MOUNT_CACHED

  /s3-external-public-by-user:
    source: s3://s3-external-public-by-user-dy
    mode: MOUNT_CACHED

  /s3-external-public-not-by-user:
    source: s3://digitalcorpora
    mode: MOUNT_CACHED

  #### GCS tests ####
  /gcs-sky-managed-copy:
    name: gcs-sky-managed-copy
    source: ~/sky_logs
    store: gcs
    mode: COPY

  /gcs-sky-managed-mount:
    name: gcs-sky-managed-mount
    source: ~/sky_logs
    store: gcs
    mode: MOUNT

  /gcs-sky-managed-mount-cached:
    name: gcs-sky-managed-mount-cached
    source: ~/sky_logs
    store: gcs
    mode: MOUNT_CACHED

  /gcs-external-private-by-user:
    source: gs://gcs-external-private-by-user-dy
    mode: MOUNT_CACHED

  /gcs-external-public-by-user:
    source: gs://gcs-external-public-by-user-dy
    mode: MOUNT_CACHED

  /gcs-external-public-not-by-user:
    source: gs://gcp-public-data-sentinel-2
    mode: MOUNT_CACHED


workdir: ~/yaml

setup: |
  echo hi

run: |
  # Show workdir
  ls -l .

  # Show private/public storage copy/mount
  ls -l /s3-sky-managed-copy
  ls -l /s3-sky-managed-mount
  ls -l /s3-sky-managed-mount-cached
  ls -l /s3-external-private-by-user
  ls -l /s3-external-public-by-user
  ls -l /s3-external-public-not-by-user
  ls -l /gcs-sky-managed-copy
  ls -l /gcs-sky-managed-mount
  ls -l /gcs-sky-managed-mount-cached
  ls -l /gcs-external-private-by-user
  ls -l /gcs-external-public-by-user
  ls -l /gcs-external-public-not-by-user

  # Write files on a mounted storage with access
  date > /s3-sky-managed-mount/hellotest.txt
  date > /s3-sky-managed-mount-cached/hellotest.txt
  date > /s3-external-private-by-user/hellotest.txt
  date > /s3-external-public-by-user/hellotest.txt
  date > /gcs-sky-managed-mount/hellotest.txt
  date > /gcs-sky-managed-mount-cached/hellotest.txt
  date > /gcs-external-private-by-user/hellotest.txt
  date > /gcs-external-public-by-user/hellotest.txt

  # Confirm file was written
  cat /s3-sky-managed-mount/hellotest.txt
  cat /s3-sky-managed-mount-cached/hellotest.txt
  cat /s3-external-private-by-user/hellotest.txt
  cat /s3-external-public-by-user/hellotest.txt
  cat /gcs-sky-managed-mount/hellotest.txt
  cat /gcs-sky-managed-mount-cached/hellotest.txt
  cat /gcs-external-private-by-user/hellotest.txt
  cat /gcs-external-public-by-user/hellotest.txt

Sheth and others added 30 commits February 16, 2024 23:04
initial commit
7dfcba1
@shethhriday29
Merge branch 'skypilot-org:master' into master
212df29
newline
a08c034
comments
b85cbf9
run linter
4fbfe17
reminder for down
6fc77e1
tentatively done with example
d6cb993
formatting
2d5aceb
yapf
4e1954a
@shethhriday29
Merge branch 'skypilot-org:master' into master
eb22f62
@romilbhardwaj
[Storage] Storage mounting tool permissions fix (skypilot-org#3215)
27a8905 
* fix permissions

* fix permissions
@Michaelvll @concretevitamin
[LLM] Example for Serving Gemma (skypilot-org#3207)
41a63df 
* Add serve for gemma and fix mixtral dependency

* Add hf token

* fix model len

* Add comment

* Serve your private gemma

* fix serve yaml

* readme

* Remove chat completion due to the wrong template

* add readme

* Update llm/gemma/README.md

Co-authored-by: Zongheng Yang <[email protected]>

* address comments

* Update README.md

Co-authored-by: Zongheng Yang <[email protected]>

* Update llm/gemma/README.md

Co-authored-by: Zongheng Yang <[email protected]>

* Update llm/gemma/README.md

Co-authored-by: Zongheng Yang <[email protected]>

* Update llm/gemma/README.md

Co-authored-by: Zongheng Yang <[email protected]>

* Change to it

* Add chat API

* use HF_TOKEN env

* typo

---------

Co-authored-by: Zongheng Yang <[email protected]>
@Michaelvll
[LLM] Add logo for Gemma (skypilot-org#3220)
2b17e91
@Michaelvll
Minor fixes for release 0.5.0 (skypilot-org#3212)
b326d12 
* when removing cudo credential, sky check fails

* remove tips

* minor hint fix

* fix cluster version for k8s

* fix typo
@Michaelvll
[Docker] Add retry for docker pull due to daemon not ready (skypilot-…
6d77872 
…org#3218)

* Add retry for docker pull due to daemon not ready

* longer wait time

* longer wait time

* retry earlier

* add retry for retries as well

* longer wait time

* change wait time

* format

* Add comment

* Fix

* Fix indent for azure docker config

* Fix docker login config

* Fix comments

* More robust docker login config

* Add retry for docker check

* minor fix

* Add additional test for stop and start with docker

* Fix cancelled
added comments
cb695d5
@shethhriday29
Merge branch 'skypilot-org:master' into master
94221eb
quick fix
888f1a8
finished pip issues
4877669
fix
7a208dd
@romilbhardwaj
fix storage error message, add example link to docs
152e36a
@romilbhardwaj
Merge branch 'master' of https://github.com/skypilot-org/skypilot int…
cf556ed 
…o hriday/unsloth

# Conflicts:
#	README.md
#	docs/source/index.rst
@shethhriday29
Merge branch 'skypilot-org:master' into master
8213f02
@shethhriday29
Merge branch 'skypilot-org:master' into master
60a530d
@shethhriday29
Merge branch 'skypilot-org:master' into master
8df52d6
@shethhriday29
Merge branch 'skypilot-org:master' into master
31291f4
prototype for gcs
c868494
removed non git files
953b4a6
rclone support for aws
e63e943
minor formatting fix
a4ddd48
@landscapepainter landscapepainter marked this pull request as draft November 15, 2024 10:08
@romilbhardwaj
Copy link
Collaborator

This is great, thanks @landscapepainter! I noticed this is only S3/GCS. Do we plan to support Azure as well in this PR?

romilbhardwaj
romilbhardwaj reviewed Dec 2, 2024
View reviewed changes
Copy link
Collaborator

@romilbhardwaj romilbhardwaj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is very cool @landscapepainter! Left some quick comments.

sky/data/data_utils.py
"""Provides methods to manage and generate Rclone configuration profile."""

class RcloneStores(enum.Enum):
S3 = 'S3'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add R2 as well? Should be very similar to S3 except with a different endpoint.

sky/data/mounting_utils.py Outdated
#f'--log-file {log_file_path} --log-level DEBUG ' #log related flags
# '--dir-cache-time' specifies the frequency of how often rclone should
# check the backend storage for an update when there is a discrepancy.
'--allow-other --vfs-cache-mode writes --dir-cache-time 30s '
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we use --vfs-cache-mode full to also cache reads?

@landscapepainter landscapepainter marked this pull request as ready for review December 27, 2024 00:50
@landscapepainter landscapepainter changed the title Mount cached mode [Storage]Mount cached mode Dec 27, 2024
@landscapepainter
Copy link
Collaborator Author

@romilbhardwaj @cblmemo @Michaelvll This is ready for a look :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@romilbhardwaj romilbhardwaj Awaiting requested review from romilbhardwaj

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@landscapepainter @romilbhardwaj @shethhriday29 @Michaelvll